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Abstract 

Under  the  direction  of  the  Principal  Investigators  and  support  from  the  Acquisition  Research  Program  and 
government  and  industry  partnerships,  the  Systems  Development  &  Maturity  Laboratory  (SysDML)  at 
Stevens  Institute  of  Technology  has  successfully  developed,  tested,  and  implemented  a  system  maturity 
measure  (i.e..  System  Readiness  Level)  and  supporting  optimization  models  for  inclusion  in  a  System 
Earned  Readiness  Management  methodology.  Currently  the  SysDML  is  integrating  these  previous 
developments  into  tools  to  quantify  maturity  in  multi-function,  multi-capability  (MFMC)  systems.  Given 
that  the  trend  is  for  systems  to  provide  MFMC,  it  has  become  challenging  for  managers  to  properly  assess 
their  development  and  acquisition  to  ensure  the  achievement  of  critical  capabilities  and  functions. 
Moreover,  such  a  challenge  is  compounded  when  the  systems  are  not  only  comprised  of  MFMC  but  have 
multiple  or  competing  technology  and  integration  alternatives.  This  challenge,  then,  raises  a  fundamental 
question:  How  do  we  effectively  assess  the  maturity  of  a  system  for  acquisition  when  considering 
technology  and  integration  alternatives  or  trade-offs  in  a  MFMC  system?  This  research  intends  to 
answer  this  question  and  provide  results  that  can  be  used  to  evaluate  systems  development  maturity, 
track  progress,  identify  component  criticality,  and  form  corresponding  strategies  for  further  development 
and  trade-offs  in  technology  and  integration  alternatives. 

This  material  is  based  upon  work  supported  by  the  Naval  Postgraduate  School  Acquisition  Research  Program  under 
Grant  No.  N00244-11-1-0002. 
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1.  Introduction 

Under  the  direction  of  the  Principal  Investigators  (Pis)  and  with  support  from  the  Naval  Postgraduate 
School  (NPS)  Acquisition  Research  Program  and  government  and  industry  partnerships,  the  Systems 
Development  and  Maturity  Laboratory  (SysDML)  at  Stevens  Institute  of  Technology  has  successfully 
done  the  following: 

a)  Developed  a  methodology  for  determining  a  system's  development/acquisition  maturity  (i.e., 
System  Readiness  Level  [SRL];  Sauser  et  al.,  2008;  Sauser  &  Ramirez-Marquez,  2009)  (NPS  BAA 
07-002,  2007); 

b)  Formulated  two  optimization  models  for  allocating  resources  so  as  to  optimize  cost,  schedule, 
and  maturity  performance  (i.e.,  SRLmax  and  SCODmin;  Sauser  &  Ramirez-Marquez,  2009;  Sauser  & 
Ramirez-Marquez,  2009;  Magnaye,  Sauser  et  al.,  2010)  (NPS  BAA  07-002,  2007); 

c)  Defined  a  methodology  that  combines  previous  Items  a  and  b  into  an  approach  called  Systems 
Earned  Readiness  Management  (SERM;  Magnaye,  Sauser,  &  Ramirez-Marquez,  2009;  Sauser  & 
Ramirez-Marquez,  2009;  Magnaye  et  al.,  2010)  (NPS  BAA  08-004,  2008);  and 

d)  Developing  models  to  determine  which  components  (i.e.,  the  technologies  are  integrations  the 
system  is  build  off)  are  sufficient,  critical,  or  important  to  achieve  a  level  of  system  maturity  for 
the  intended  functionalities  and  capabilities  that  can  satisfy  warfighters'  needs  (Tan  et  al., 
2010a;  Tan  et  al.,  2010b)  (NPS  BAA  09-002,  2009). 

It  is  important  to  note  that  throughout  these  research  efforts,  the  SysDML  has  maintained  an 
evolutionary  development  approach  through  which  we  have  worked  closely  with  industry  and 
government  to  refine  and  implement  our  research  in  order  to  ensure  its  relevance  and  rigor  (see  Exhibit 
1). 

The  research  results  described  in  this  report  address  some  recurring  issues  that  were  revealed  through 
conversations  with  our  industry  and  government  research  partners.  In  essence,  we  are  answering  the 
following  question:  What  are  the  effects  of  necessary  trade-offs  in  functionality,  capability,  cost, 
schedule,  and  maturity  that  will  allow  the  acquisition  of  a  system  that  can  still  satisfy  warfighter  needs? 
The  objective  is  to  be  able  to  do  the  following: 

1)  Identify  the  critical  components  to  system  maturity;  that  is,  which  components  (i.e., 
technologies  or  integrations)  have  the  greatest  impact  on  system  maturity; 

2)  Prioritize  component  development  based  on  constrained  resource  availability;  and 

3)  Balance  between  system  capabilities/functions  with  a  given  developmental  budget. 
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Exhibit  1.  Evolutionary  Development  Plan 


While  the  previous  research  is  absolutely  necessary  and  relevant  for  a  better  understanding  of  how  to 
achieve  effective  maturity,  capability,  and  functionality  of  a  system  for  acquisition,  it  still  does  not 
address  a  fundamental  activity  in  the  development  of  system  solutions  that  this  research  addressed.  To 
clarify,  in  any  systems  solution,  a  systems  engineer  or  acquisition  manager  must  make  critical, 
necessary,  and  sufficient  trade-offs  with  respect  to  technologies  and  integrations  based  on  Analysis  of 
Alternatives  (AoA),  and  these  trade-offs  come  at  a  cost/benefit  to  the  functionality  and  capability  of  the 
system  and  its  existing  architecture.  There  is  increasing  development  and  acquisition  of  systems  that 
are  built  upon  open  and  flexible  platform  designs  that  can  accommodate  multiple  functions  and 
capabilities  as  well  as  the  ability  to  adopt  future  mission  packages  (e.g.,  modularity,  system  of  systems). 
With  such  a  trend,  decisions  to  trade  off  among  multiple  alternatives  that  enable  necessary  functions 
and  capabilities  will  be  unavoidable.  Therefore,  this  research  is  timely  and  critical  in  that  it  intends  to 
build  upon  the  current  research  funded  by  the  Acquisition  Research  Program  on  System  Capability 
Satisficing  to  develop  a  methodology  for  an  AoA  (or  trade-off)  for  technology  and  integration 
component  adoption,  which  impacts  the  system  maturity,  functionality,  and  capability  (see  the  orange 
arrow  in  Exhibit  1). 
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2.  Purpose  and  Focus  of  Research 

Over  the  past  four  years,  the  SRL  scale  and  supporting  methods  have  undergone  continuous 
development  and  have  been  accepted  as  a  valid  metric  to  measure  the  readiness  of  a  system 
throughout  its  development  life  cycle  (implementations  of  this  metric  have  been  performed  by  U.S. 
Navy-PMS420,  U.S.  Army-ARDEC,  Lockheed  Martin,  and  Northrop  Grumman).  However,  to  date,  this 
scale  has  focused  on  the  management  of  systems  intended  to  perform  a  single  function.  Today,  even  the 
most  basic  weapons  such  as  assault  rifles  have  become  multi-functional.  As  such,  during  the 
development  process,  these  systems  may  be  called  upon  to  deliver  some  of  their  capabilities  even  as  the 
development  of  others  is  still  several  phases  behind.  Often,  this  requires  an  AoA  among  technology 
choices  and  architectures  to  take  advantage  of  those  that  are  already  mature,  though  not  originally 
intended  for  use  in  the  development  of  the  desired  function.  This,  in  turn,  requires  a  thorough 
understanding  of  the  technical  aspects  of  the  components  but,  more  significantly,  the  relative 
importance  of  each  choice  on  the  readiness  of  the  system  vis-a-vis  its  desired  capability. 

One  of  the  first  researchers  to  describe  this  necessity  for  the  management  of  systems  development  was 
Birnbaum  (1969),  who  introduced  a  quantitative  definition  of  component  importance.  Since  then,  the 
area  of  component  Importance  Measures  (IM)  has  received  significant  attention,  and  its  utility  has 
gained  more  and  more  recognition.  For  example,  Hwang  (2001)  specified  that  an  importance  index  is  for 
the  measurement  of  the  relative  importance  of  a  component  with  respect  to  other  components  in  a 
system,  and  thus,  it  can  be  used  to  rank  the  contributions  of  components  or  basic  events  to  the  system's 
performance  (Baraldi,  Zio,  &  Compare,  2009).  Thus,  by  their  definition,  IM  can  quantify  the  criticality  of 
a  particular  component  within  a  system,  be  used  as  a  tool  for  evaluating  and  ranking  the  impact  of 
individual  components,  determine  the  most  important  component  with  respect  to  the  overall  system 
performance,  identify  system  weaknesses,  and  prioritize  system  performance  improvement  activities 
(Ramirez-Marquez  and  Coit,  2005,  2007).  Zio,  Marella,  and  Podofillini  (2007)  further  emphasize  the  great 
practical  aid  provided  by  IMs  in  that  they  allow  system  designers  and  managers  to  track  system 
bottlenecks  and  also  provide  guidelines  for  effective  actions  when  determining  system  improvements. 
This  body  of  knowledge  in  IM  research  has  added  significant  value  to  our  understanding  of  the  non¬ 
linearity  in  the  importance  of  technologies  and  integrations  within  a  system,  but  it  does  not  consider  the 
technology  and  integration  alternatives  (or  trade-offs)  and  their  importance  to  the  decisions  made  in  the 
development  and  acquisition  of  a  systems  solution. 

In  practice,  a  system  evolves  with  time  from  a  single  capability  or  a  specific  function  to  a  more 
complicated  one  that  affords  multiple  functions,  and  to  ensure  the  operational  performance  of  a 
function  by  having  several  backup  capabilities.  Kim  et  al.  (2009)  specified  that  a  technical  system  usually 
comprises  a  number  of  subsystems  and  components  that  are  interconnected  in  such  a  way  that  the 
system  is  able  to  perform  multiple  required  functions.  Moreover,  in  order  to  ensure  the  success  of  the 
development  or  acquisition  of  a  system,  even  for  a  specific  function,  it  is  common  to  have  one  or  several 
backup  capabilities.  These  systems  are  often  required  to  be  open  to  further  integration  of  other  mission 
packages  in  order  to  satisfy  future  requirements  for  a  yet-to-be-defined  service  (GAO,  2008).  In 
addition,  in  the  evolution  of  systems  development,  the  advancement  of  technology  options  is 
progressing  faster  than  the  systems  themselves.  In  this  process,  the  following  questions  may  arise:  Will 
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a  new,  more  functional  system  or  technology  supersede  the  old?  Has  the  system  or  technology  become 
inadequate  due  to  changes  in  other  systems  or  technologies?  Is  it  more  effective  to  invest  in  the 
development  of  a  new  system  or  technology?  Has  the  system  or  technology  lifetime  been  shortened  by 
recent  developments?  What  is  a  robust  methodology  that  can  effectively  and  efficiently  analyze, 
compare,  and  trade  off  technology  alternatives? 

With  an  ever-increasing  complexity  of  the  systems  that  are  being  developed  and  realized,  multiple 
functions  and  capabilities  are  common  to  the  development  of  most  systems,  which  raises  the  need  to 
balance  development  objectives  when  facing  multiple  component  alternatives.  As  a  result,  managers 
require  metrics  that  enable  the  assessment  of  multi-function,  multi-capability  (MFMC)  system 
developmental  maturity  to  manage  the  potential  risks  posed  by  the  previous  questions  (Volkert,  2009). 

For  example,  Forbes,  Volkert,  Gentile,  and  Michaud  (2009)  have  described  a  system  with  six  capabilities 
that  are  realized  by  six  threads  of  components.  The  architecture  of  this  system  is  represented  in  Exhibit 
2.  As  described  in  their  paper,  this  system  had  undergone  a  system  maturity  assessment  with  summary 
charts  also  depicted  in  Exhibit  2.  The  initial  system  architecture  represented  in  Exhibit  2(a)  resulted  in 
an  assessment  with  an  overall  SRL  of  0.60  (see  MP  SRL  in  the  upper  right  box  of  the  diagram);  the 
identification  of  an  insufficiently  mature  technology  and  supporting  integrations  (circled  in  the  diagram); 
and  analysis  of  the  technology-integration  maturity  of  all  the  system  components  (a  horizontal  line  at 
the  bottom  of  the  diagram).  An  alternative  systems  solution  based  on  a  trade-off  (see  Exhibit  2[b]), 
considered  replacing  a  single  technology  (i.e.,  MVCS)  with  another  two  technologies  (i.e.,  DLS  OB;  DLS 
RMMV).  This  alternative  does  not  significantly  improve  the  overall  SRL  value  (an  increase  of  only  0.04), 
but  it  does  improve  the  technology-integration  maturity  of  all  the  lagging  system  components  (a 
horizontal  line  at  the  bottom  of  the  diagram).  While  this  analysis  may  seem  sufficient  in  increasing 
systems  maturity  toward  an  effective  acquisition  decision,  it  does  not  consider  many  of  the  IMs  related 
to  an  AoA  and  their  influence  on  the  system's  current  or  future  maturity.  Thus,  a  decision  made  purely 
on  an  increase  in  maturity  may  be  insufficient.  This  described  research  intends  to  address  this  concern 
by  enhancing  the  SRL  approach  to  take  into  account  the  multiple  system  functions  and  capabilities, 
which  allows  for  the  opportunity  to  better  understand  an  analysis  of  alternatives  in  technologies  and 
integrations  to  more  effectively  manage  system  maturity  and  acquisition. 
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Exhibit  2.  Technology/Integration  Trade-Off  Analysis 


3.  Research  Approach 

In  the  domain  of  system  architecture  and  AoA,  identifying,  prioritizing^  and  ranking  components  with 
respect  to  their  impact  on  the  system  is  key  to  understanding  the  importance  of  any  one  component  to 
another  and  is  notable  for  suggesting  trade-offs  among  key  parameters  during  their  development  and 
acquisition.  This  research  contributes  to  the  body  of  knowledge  by  bridging  the  gap  in  the  analysis  of 
traditional  system  architectures  with  system  maturity  assessment.  It  is  expected  that  such  analysis  will 
assist  in  the  establishment  of  system  development  and  acquisition  strategies  from  the  quantification  of 
the  relationship  between  component  maturity  and  system  maturity  and  distinguishing  the  importance 
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of  different  components  for  making  informed  decisions  when  deciding  to  trade  off  technology  and 
integration  alternatives  without  rushing  to  judgment  on  a  preferred  systems  solution  or  architecture. 

In  an  AoA,  the  main  objective  is  to  execute  an  extensive  analysis  that  would  result  in  a  preferred  system 
architecture.  Key  in  the  execution  of  this  analysis  will  be  the  IMs  that  describe  the  most  important 
components  for  the  performance  of  the  system.  What  we  are  proposing  is  a  variation  on  a  traditional 
AoA  in  that  the  key  performance  indicator  would  be  the  maturity  of  the  technologies  and  integrations 
supporting  the  systems'  functionalities  and  capabilities  and  the  yet-to-be-defined  IMs.  To  enable  the 
AoA  by  taking  into  consideration  the  notions  of  function  and  capability  in  a  system,  this  research 
proposes  a  hierarchical  SRL  (see  Exhibit  3),  where  the  SRL  is  defined  at  the  following  three  different 
levels: 

1)  Capability-based  SRL 

SRL_Cfk denotes  the  SRL  for  capability  Cfk.  It  is  defined  as  the  average  of  all  the  normalized  Integrated- 
Technology  Readiness  Level  (ITRL)  values,  which  is  given  by 


SRL  Cfk  = 


ITSL_Cm  +  ITRL_CM2,  +  ...+ITRL_Cfi',>  'LITRL-Cm 


n 


A 


n 


A 


2)  Function-based  SRL 


SRL_Ff  is  the  SRL  for  function  Ff.  Although  there  are  multiple  capabilities  to  ensure  the  same  function, 
the  maximum  of  these  Capability-based  SRLs  represents  the  readiness  of  that  function  and  is  defined  as 

SRL_Ff  =  Max(SRL  _  CJk),  k  =  1,2 

SRL_F  matrix  includes  all  the  Function-based  SRLs  and  is  denoted  by 


SRL  F  = 


SRL_Fl 

SRL_F2 


(10) 


SRL  _Fr 

3)  Composite  SRL 

Composite  SRL  for  the  whole  system  is  the  average  of  all  Functionality  SRLs  and  is  defined  as 


(ZsRL_Fu)  +  <'£sRL_F2t)  +  ...+<ZsRL_Frk) 
Composite  SRL  -  - 


k= 1 _ 

Kx+K2+...+Kr 


k= 1 


/= 1  k= 1 
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Exhibit  3.  SRL  Hierarchy 


The  capability-based  SRL  calculates  the  SRL  for  a  particular  capability  thread  that  includes  a  set  of 
components  to  enable  an  intended  capability.  Based  on  the  calculation  of  SRL_C's,  the  function-based 
SRL  addresses  the  SRL  for  a  specific  function  that  encompasses  one  or  several  capability  threads.  The 
Composite  SRL  indicates  the  SRL  for  the  whole  system,  which  includes  multiple  functions  with  multiple 
capabilities.  This  hierarchical  assessment  is  then  used  as  a  baseline  for  the  AoA  related  to  trade-offs  in 
technology  and  integration  options.  Exhibit  4  represents  a  typical  process  of  an  AoA.  The  phases 
identified  in  Exhibit  4  are  described  in  the  next  section. 
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Exhibit  4.  Alternative  Analysis 

This  research  was  executed  in  the  following  three  phases: 

Phase  1— Identify  MFMC  AoA  Metrics:  This  phase  will  identify  key  metrics  for  the  development  of  the 
MFMC  AoA  IMs  as  it  relates  to  a  trade-off  analysis.  What  we  have  proposed  are  preliminary 
representations,  which  need  further  work  and  understanding  on  how  they  relate  to  architectural 
variants  in  functionality  and  capability  for  acquisition. 

Phase  2— Verification  of  the  MFMC  AoA  Metrics  and  Models  with  Systems:  We  applied  these  models  to 
real  systems  from  Lockheed  Martin  (Morristown,  NJ;  Orlando,  FL),  Northrop  Grumman  Integrated 
Systems,  and  the  U.S.  Army  Armament  Research,  Development  and  Engineering  Center  (ARDEC).  This 
application  resulted  in  refinement  and  adjustment  to  the  quantitative  correlations  of  the  models  based 
on  Phases  1  and  2. 

Phase  3— Development  of  a  Methodology:  We  documented  a  process  methodology  for  the  use  of  the 
MFMC  AoA  models  so  further  work  can  be  carried  out  with  validation  of  the  models  with  those 
stakeholders  identified  in  Phase  2.  We  also  explored  some  preliminary  optimization  models  and 
capability  satisficing  indexes  for  development  of  a  more  robust  acquisition  decision  approach. 

4.  illustrative  Example  of  Implementation  of  Results 

The  MFMC  AoA  methodology  is  demonstrated  with  a  system  that  was  previously  discussed  by  Forbes, 
Volkert,  Gentile,  and  Michaud  (2009)  and  shown  in  Exhibit  2.  There  are  basically  two  functions— Mine 
Detection  and  Mine  Neutralization— to  be  performed  by  this  system.  There  are  six  capabilities  that  are 
realized  by  six  threads  of  components,  as  listed  in  Exhibit  5:  four  capabilities  in  the  first  function  (shaded 
green  in  the  exhibit)  and  two  capabilities  in  the  second  function  (shaded  yellow  in  the  exhibit).  These 
capabilities  are  1)  Bottom  Mapping  &  Change  Detection;  2)  Shallow  &  Littoral  Water  Mine  Detection;  3) 
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Bottom  &  Volume  Mine  Detection-1;  4)  Bottom  &  Volume  Mine  Detection-ll;  5)  Contact  Mine 
Neutralization;  and  6)  Influence  Mine  Neutralization. 


(1)  Bottom  Mapping  &  Change  Detection  thread 
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(2)  Shallow  &  Littoral  Water  Mine  Detection  thread 
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(3)  Bottom  &  Volume  Mine  Detection  I  thread 
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(5)  Contact  Mine  Neutralization  thread 


(6)  Influence  Mine  Neutralization  thread 


Exhibit  5.  A  MFMC  System  with  Two  Functions  and  Six  Capabilities 

Exhibit  6  shows  the  results  from  using  a  tool  that  utilizes  the  proposed  SRL  hierarchy  to  assess  system 
maturity.  The  left  side  of  the  figure  displays  the  new  hierarchy  for  the  estimates  of  system  development 
maturity  at  different  levels:  ITRL,  capability,  function,  and  system  levels.  It  adds  two  layers  (capability 
and  function)  to  the  original  SRL  definition  to  provide  more  accurate  assessment  and  more  insights  for 
systems  engineering  managers  to  track  the  progression  on  life  cycle  of  the  systems'  development.  In 
addition,  three  other  user  inputs  are  added  to  the  assessment: 


•  Expected  SRL:  this  is  the  SRL  value  that  is  expected  at  the  time  of  the  assessment  or  a  projected 
time  in  the  life  cycle. 

•  Red  Bar:  this  is  the  lower  threshold  value  of  the  SRL.  If  an  ITRL,  capability,  or  function 
assessment  falls  below  this  level,  it  is  indicated  in  red. 
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•  Yellow  Bar:  this  is  the  upper  threshold  value  of  the  SRL.  If  an  ITRL,  capability,  or  function 
assessment  falls  above  this  level,  it  is  indicated  in  yellow. 


Any  value  for  the  ITRL,  capability,  or  function  assessment  that  falls  within  the  thresholds  is  indicated  in 
green.  The  development  of  this  illustrative  system  is  mapped  to  the  DoD  Acquisition  Life  Cycle  to 
determine  the  development  progression.  As  shown  in  Exhibit  6,  with  a  10%  margin  of  the  expected  SRL 
0.53  as  the  risk  thresholds,  we  can  observe  the  following: 

1)  The  system  level  SRL  indicates  that  the  whole  system  is  progressing  on  schedule  with  an  SRL 
value  of  0.51,  compared  to  the  expected  value  on  this  particular  assessment  date. 

2)  Although  the  development  of  Function  1  is  ahead  of  schedule  with  an  SRL  value  of  0.59,  there  is 
variability  in  the  development  of  the  individual  capabilities  that  make  up  Function  1. 

3)  The  ITRLs  in  Exhibit  6  are  for  the  selected  Capability  2.2,  which  is  within  the  target  threshold,  but 
ITRLs  2  and  4  indicate  a  risk  in  falling  behind,  though  the  rest  the  ITRLs  in  this  capability  are 
being  developed  as  planned. 


How  to  Use  this  Template _ 

1-input  the  date;  2-move  the  selector  in  the 
System  Development  Lifecycle  chart  to  choose 
an  expected  SRL  for  this  date;  3-specify  the 
percentage  for  RED  and  GREEN  threshold;  /- 
check  the  CAPABILITY  of  interest;  5-for  ? 
scenario  analysis,  change  the  TRL/IRL  in  tfte 

-V 


1 -Combat 
Mgmt  System 
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]t  H>|*  10.usv  j«  B  1: 
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(RMMV)  T  \  C| 


ITRL  7  0.58 
ITRL  10  0.58 

ITRL  13  0.59 


Pre-Systems  Acquisition 


Systems  Acquisition 


Sustainment 


Exhibit  6.  Alternative 


The  use  of  the  enhanced  SRL  hierarchy  and  the  employed  assessment  tool  provides  developers  and 
managers  with  a  holistic  picture  to  investigate  the  system  development  and  with  the  ability  to  easily 
identify  system  development  weaknesses  as  indicated  by  variations  around  a  threshold.  This  example 
also  manifests  the  multi-dimensional  nature  of  system  maturity  assessment,  which  should  be  examined 
at  different  levels  within  the  same  system  architecture  for  which  an  assessment  using  a  single  number 
would  have  overlooked. 
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This  tool  and  approach  facilitates  the  aforementioned  AoA  to  examine  multiple  alternatives  to  get  the 
best  possible  solution  to  satisfy  customer  requirements.  Exhibit  7  shows  the  assessment  of  a  different 
alternative  to  Exhibit  6,  only  replacing  Technology  2-i  with  2-ii  that  is  assumed  to  provide  the  same 
functionality  (see  the  circled  technology).  While  more  mature  in  its  TRL  and  IRLs  with  Technologies  4  and 
6,  the  inclusion  of  Technology  2-ii  leverages  the  progression  of  Capabilities  1.1  and  1.3,  which  were 
previously  identified  to  be  lagging  in  Alternative-i  (Exhibit  6).  Meanwhile,  at  the  ITRL  level  for  Capability 
2.2,  all  ITRLs  are  balanced  to  be  either  on  or  ahead  of  development  schedule.  Compared  with 
Alternative-i,  Alternative-ii  (Exhibit  7)  significantly  outperforms  in  terms  of  meeting  and  balancing 
development  maturity  and  potentially  mitigates  some  of  the  risk  of  meeting  customer  expectations. 
Therefore,  based  on  the  analysis  of  these  two  alternatives,  the  second  option  with  Technology  2-ii  is  the 
preferred  alternative. 


ITRL 

1 

0.61 

ITRL 

2 

0.50 

ITRL 

3 

0.61 

ITRL 

4 

0.49 

ITRL 

7 

0.58 

ITRL 

10 

0.58 

ITRL 

13 

0.59 

Sustainment 


Exhibit  7.  Alternative  ii 


It  should  be  noted  that  although  it  seems  quite  simple  and  intuitive  from  the  comparison  of  these  two 
alternatives,  this  example  is  only  for  illustration  purposes.  In  practice,  the  method  presented  in  this 
paper  will  have  more  value  when  utilized  on  more  complicated  architectures  that  involve  a  number  of 
different  components  (technologies  and  integrations)  and  the  interplay  among  them.  In  such  situations, 
it  will  not  be  straightforward  and  intuitive  like  this  one  and  can  hardly  be  cognitively  comprehended, 
where  the  use  of  such  a  tool  and  multi-layer  hierarchy  is  very  necessary  for  trade-offs  in  MFMC  systems 
acquisition. 


5.  Managerial  Implications  and  Future  Research 

In  order  to  address  a  challenging  system  acquisition  problem  of  effectively  assessing  the  development 
maturity  with  the  consideration  of  technology  and  integration  alternatives  in  MFMC  systems,  this 
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research  proposes  an  enhanced  SRL  hierarchy  and  an  AoA  methodology.  With  the  use  of  a  tool,  the 
proposed  methodology  was  demonstrated  with  an  illustrative  example.  As  evidenced,  this  methodology 
moves  us  closer  to  facilitating  more  informed  maturity  assessments  for  MFMC  systems  development 
and  AoA  that  can  assist  the  acquisition  of  DoD  weapon  systems. 

Previous  efforts  funded  by  the  Acquisition  Research  Program  addressed  some  recurring  issues  that  were 
revealed  through  conversations  with  our  industry  and  government  research  partners.  In  essence,  we 
were  answering  the  following  question:  What  are  the  effects  of  necessary  trade-offs  in  functionality, 
capability,  cost,  schedule,  and  maturity  that  will  allow  the  acquisition  of  a  system  that  can  still  satisfy 
warfighters'  needs?  Our  research  funded  via  the  Acquisition  Research  Program  helped  to  answer  the 
question  by  1)  identifying  the  critical  components  to  system  maturity;  that  is,  which  components  (i.e., 
technologies  or  integrations)  have  the  greatest  impact  on  system  maturity;  2)  prioritizing  component 
development  based  on  constrained  resource  availability;  and  3)  balancing  between  system 
capabilities/functions  with  a  given  developmental  budget. 

While  the  previous  research  has  proven  necessary  and  relevant  for  a  better  understanding  of  how  to 
achieve  effective  maturity,  capability,  and  functionality  of  a  system  for  acquisition,  it  does  not  address  a 
fundamental  activity  in  the  development  of  system  solutions  that  this  research  introduces.  To  clarify,  in 
any  systems  solution,  a  systems  engineer  or  acquisition  manager  must  make  critical,  necessary,  and 
sufficient  trade-offs  with  respect  to  technologies  and  integrations  based  on  AoA,  and  these  trade-offs 
come  at  a  cost/benefit  to  the  functionality  and  capability  of  the  system  and  its  existing  architecture. 
There  is  increasing  development  and  acquisition  of  systems  that  are  built  upon  open  and  flexible 
platform  designs  that  can  accommodate  multiple  functions  and  capabilities  as  well  as  the  ability  for 
adopting  future  mission  packages  (e.g.,  modularity,  system  of  systems).  With  such  a  trend,  decisions  to 
trade  off  among  multiple  alternatives  that  enable  necessary  functions  and  capabilities  will  be 
unavoidable. 

6.  Knowledge  Transfer  to  Industry/Government 

We  have  continued  to  maintain  a  rich  knowledge  transfer  relationship  with  industry  and  government. 
The  following  are  some  examples: 

•  U.S.  Army  Armament  Research  Development  and  Engineering  Center  (ARDEC):  The  ARDEC  has 
implemented  the  SRL  method  on  an  innovative  lightweight  vehicle  program  that  has  allowed 
them  to  assess  current  development  risks  as  well  as  create  development  plans  for  program 
reviews. 

•  Lockheed  Martin  (LMCO):  LMCO  has  piloted  the  SRL  on  four  projects.  Based  on  its  success  and 
lessons  learned  from  the  application,  they  are  now  using  it  on  a  major  weapons  program.  In 
addition,  they  used  internal  corporate  funds  to  create  an  SRL  software  tool  that  has  allowed 
them  to  calculate  an  SRL  and  evaluate  development  risks  for  program  reviews. 

•  Northrop  Grumman  (NGC):  NGC  has  continued  to  use  SRL  in  support  of  their  efforts  with 
PMS420  and  incorporated  the  methods  into  their  own  internal  tool  for  program  assessment. 
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7.  Project  Accomplishments 

7.1.  Publications 

7. 1. 1.  Journal 

Concho,  L.,  Ramirez-Marquez,  J.,  Hearld,  T.,  &  Sauser,  B.  (2011).  Functionally  equivalent  COTS  for 
optimal  component  substitution  within  system  evolution  planning.  Technology  Analysis  & 
Strategic  Management,  23(5),  509-526. 

7.1.2.  Conference  Proceedings 

Sauser,  B.,  Tan,  W.,  &  Ramirez-Marquez,  J.  (2011,  May  11-13).  Analysis  of  alternatives  in  system 
capability  satisficing  for  effective  acquisition.  In  Proceedings  of  the  Eighth  Annual  Acquisition 
Research  Symposium.  Monterey,  CA:  Naval  Postgraduate  School. 

7.1.3.  Working  Papers 

Sarfaraz,  M.,  &  Sauser,  B.  (2011).  Improving  system  maturity  assessment  using  system  engineering 
architectures. 

Magnaye,  R.,  Sauser,  B.,  &  Pantanakul,  P.  (2011).  Earned  readiness  management  for  scheduling, 
monitoring  and  evaluating  the  development  of  systems. 

Sauser,  B.  J.,  Magnaye,  R.,  Tan,  W.,  Ramirez-Marquez,  J.,  &  Sauser,  B.  W.  (2011).  Optimization  of 
system  maturity  and  equivalent  system  mass  for  space  systems  engineering  management. 

Tan,  W.,  Sauser,  B.,  Ramirez-Marquez,  J.,  &  Magnaye,  R.  (2011).  Multi-objective  optimization  in 
multifunction  multicapability  systems  development  planning. 

7.2.  Presentations 

Sauser,  B.,  &  Ramirez-Marquez,  J.  (2010,  May  19).  System  capability  satisficing  in  defense 
acquisition  via  element  importance  measures.  Paper  presented  at  the  Seventh  Annual 
Acquisition  Research  Symposium.  Monterey,  CA:  Naval  Postgraduate  School. 

8.  Other  Related  Activities 

8.1.  Website 

From  the  birth  of  this  research,  we  have  believed  in  an  open  academic  model  of  sharing  our  research 
outcomes  in  the  broadest  sense  possible.  Thus,  we  have  sustained  our  website, 
http://www.SysDML.com,  for  the  distribution  of  our  research  results.  At  this  website,  you  will  find  the 
following  pages:  Research  Overview;  Publications;  Research  Projects;  Tools;  and  Who  We  Are. 

8.2.  Student  Research  Supported/Supervised 

Our  funding  from  the  Naval  Postgraduate  School  has  afforded  us  the  privilege  to  support  one  PhD 
student  to  assist  in  the  execution  of  this  research.  It  has  also  allowed  us  the  ability  to  attract  graduate 
students  to  pursue  related  and  supportive  research.  These  students  are  the  following: 
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Ivonne  Donate.  PhD  student  (currently  supported  by  the  Department  of  Defense).  Evolutionary 
lifecycle  assessment  for  disruptive  technology  integration. 

Ryan  Gove.  PhD  student  (currently  supported  by  Lockheed  Martin).  Model  based  systems 
engineering  for  effective  system  maturity  assessment. 

Samuel  Russell.  PhD  student  (currently  supported  by  NASA).  The  thermodynamics  of  system 
development:  A  mechanistic  approach  to  system  maturity  assessment. 

Matin  Sarfaraz.  PhD  student  (currently  supported  by  the  Innovation  and  Entrepreneurship  Graduate 
Fellowship).  Systems  engineering  artifact  correlation  to  systems  maturity  assessment. 

Weiping  Tan.  PhD  student  (currently  supported  by  NPS).  Methodologies  for  component  importance 
analysis  in  multi-function,  multi-capability  system  developmental  maturity  assessment. 

Joseph  Uzdzinski  (currently  supported  by  Lockheed  Martin).  System  maturity  assessment  for 
dynamic  interoperability  in  complex  systems. 

Graduates 

Romulo  Magnaye.  (May  2011).  Using  a  system  maturity  scale  to  monitor  and  evaluate  the 
development  of  complex  systems.  PhD  in  Engineering  Management.  Currently  an  Assistant 
Professor  at  Ramapo  College. 

8.3.  Student  Projects  Supervised 

Within  the  School  of  Systems  and  Enterprises  at  Stevens  Institute  of  Technology,  students  are 

encouraged  to  complete  a  3-credit  special  problems  project  as  part  of  their  course  requirements. 

Because  of  the  success  of  this  research,  we  have  been  able  to  attract  a  number  of  students  to  pursue 

projects  related  to  SRL  and  related  topics.  Here  is  a  list  of  these  students  and  projects: 

Legath,  M.  (2011).  System  lifecycles  and  the  unpredictable  nature  from  conception  to  sustainment. 
MS  Special  Problems  in  Systems  Engineering. 

Tay,  K-H.  (2011).  Addressing  obsolescence  in  the  system  readiness  level  (SRL)  scale  to  support  major 
defense  acguisition  decisions.  MS  Special  Problems  in  Systems  Engineering. 
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